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In November 2004, economists Thomas Fuchs and Ludger Woessmann pub- 
lished a statistical analysis of the relationship between technology and student 
achievement using year 2000 data from the Programme for International Student 
Assessment (PISA). The 2000 PISA was the first in a series of triennial assessments 
of 1 5-year-olds conducted by the Organisation for Economic Co-operation and 
Development (OECD). The assessment included problems in reading, math, and 
science, as well as questions about student background, school characteristics, and 
information on the use of computers and the Internet at home and at school. Fuchs 
and Woessmann analyzed data from 31 countries: 96,855 students tested in math 
and 174,227 students tested in reading. (In these same countries 96,785 students 
were tested in science, but Fuchs and Woessmann chose to concentrate on math 
because of its relation to future job performance and its relative comparability across 
countries, and on reading because it had special emphasis in the 2000 PISA.) 

Initial analyses of PISA data in 200 1 had indicated that there was a signifi- 
cant positive relationship between academic achievement and computer access. 
However, when Fuchs and Woessmann controlled for family-background char- 
acteristics using multivariate regressions, they found: 

• no significant effect on math and reading for computer access in 
school. 

• a negative effect on achievement for computer access at home. 

• a positive effect for Internet use and educational software at home. 

• a “conditional” positive effect for computer and Internet use at school 
(positive at moderate frequencies of use, but negative for low and high 
levels of use) [Fuchs & Woessmann, 2004] . 

These findings set off a flurry of headlines in both the popular and education 
press: “Too Much Computer Exposure May Hinder Learning” (USA Today 
[MacDonald, 2004]); “Doubts About School Computer Use” (BBC News 
[2004]); “Toxic Computers?” (NEA Today [2005]); “Do Computers Help with 
Children’s Education?” (ZDNet Australia [Lemay, 2005]); “Effect of Com- 
puter Use on Student Achievement Unclear” (TechLearning [2004]). Given the 
implications of this kind of exposure for educational policy, it seems essential 
to ask certain questions — ^What exactly did the study examine? How were the 
data analyzed? What exactly did the analysis find? — before educational decision 
makers can respond to the news bites or consider how to use such a study to 
guide policy and practice. 
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WHAT EXACTLY DID THE STUDY EXAMINE? 

The PISA database consisted of several types of information: 

• Scores for 15-year old students in literacy, science, and math. As noted 
above, Fuchs and Woessmann concentrated on math and reading. They 
did not disaggregate scores by country. According to the OECD, the 
content of the PISA assessment is designed “to measure how far young 
adults. . . near the end of compulsory schooling are prepared to meet the 
challenges of today ‘s knowledge societies. The assessment is forward- 
looking, focusing on young people’s abilities to use their knowledge and 
skills to meet real-life challenges, and rather than on the extent to which 
they have mastered a particular school curriculum” (Organisation for 
Economic Co-operation and Development, 2002, p. 9). 

• Survey questions of the students about their home and school environ- 
ments. These included questions about the types of appliances (includ- 
ing computers) in the house, parents’ occupations and education, the 
kinds of discussions children have with their parents, what classes the 
students are taking, students’ attitudes toward teachers’ instruction and 
classroom management, the frequency of use of various school resourc- 
es (including computers and the Internet), and various other aspects of 
home and school life. 

• Survey questions of the students’ principals about their school environ- 
ments. These included questions about the school’s resources (includ- 
ing access to computers), the number of teachers in the school and 
their qualifications, characteristics of the student body, the relationship 
between the school and students, administrative structures, and teach- 
ing practices. 

• Statistical data, such as the gross domestic product for the school’s 
country. 

The specific questions relating to technology that Fuchs and Woessman exam- 
ined appear in Table 1, along with the source of the information. 

For purposes of their analysis, Fuchs and Woessman considered home comput- 
er access to be indicated by two dichotomous dummy variables derived from the 
PISA survey. “One computer at home” or “More than one computer at home.” 
School computer access was specified by a similar pair of dummy variables derived 
from the principals’ responses: “Learning hindered a lot by lack of computers” 
or “Learning not hindered at all by lack of computers.” School access was also 
elaborated by the two computer-density measures, overall numbers of comput- 
ers and internetworked computers per student. 

Home computer use was indicated by reported frequency of reading e-mail and 
Web pages (dichotomous dummies “Never or hardly ever” or “Several times 
a week”), Internet access at home, and having educational software at home. 
School computer use was indicated by frequency of computer use and of Internet 
use (“Never or hardly ever” or “Several times a week” for both variables). 

Because their hypothesis was that school and family characteristics would 
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Table 1. Technology questions on the PISA 2000 
Student and Principal Questionnaires. 


Question text in English 

Response options 

Asked of: 

In your home, do you have 

Yes/No 

Students 

educational software? 



In your home, do you have a 



link to the Internet? 



How many of these do you have 

None / 1 / 2 / 3 or more 

Students 

in your home: Computer? 



How often do you read these 

Never or hardly ever 

Students 

materials because you want to\ 

A few times a year 


E-mail, Web pages? 

About once a month 


At your school, how often do you 

Several times a month 


use computers? 

Several times a week 


At your school, how often do you 



use Internet? 



In your school, how much is the 

Not at all 

Principals 

learning of 1 5-year-old students 

Very little 


hindered by not enough 

To some extent 


computers for instruction? 

A lot 


In your school, about how many 

(Enter number) 

Principals 

computers are: in the school 



altogether? 



connected to the Internet/ 



World Wide Web? 




interact with the technology variables to predict outcomes, Fuchs and Woess- 
mann generated several clusters of control variables based on demographics 
and other information. These included student characteristics (gender, age, and 
grade); family background (28 socioeconomic variables); resource inputs (a dozen 
variables including class size, teacher education, instruction time, etc.); and 
institutional characteristics (a dozen variables including the use of exit exams 
and standardized tests and various measures of school autonomy). Fuchs and 
Woessmann also created a school composition variable from the school means for 
gender and all of the family background variables and a set of country dummy 
variables. These last two sets of variables were intended to control for sources of 
statistical error attributable to relationships between socio-economic status and 
school and to systematic differences between countries. (This analysis did not 
otherwise make any distinction between countries.) 

HOW WERE THE DATA ANALYZED? 

Fuchs and Woessman proposed a “production function” model that described a 
student’s test score as a function of computer access and use plus background data 
on school and family characteristics. The model also included a certain amount of 
measurement error caused by imputing values for surveys where the respondent 
had failed to answer some of the questions. (This is a common problem on a large 
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survey where many respondents leave one or more questions blank. The cleanest 
solution is to simply drop the entire record, but that would have resulted in a loss 
of nearly 73% of the PISA data.) Finally, the model included a general error term 
for variance that could not be otherwise accounted for. 

The analysis began by considering only part of the equation — scores as a func- 
tion of computer access. Fuchs and Woessman repeated the regression analyses 
several times for both home and school access, each time accounting for the ad- 
vantages of additional control variables until they had an estimate of the effect 
of computer access that was corrected for all the other effects in their model. 

The analyses of computer use were somewhat different. Fuchs and Woess- 
mann included all the control variables from the start. They also included the 
effects of computer access. That is, they attempted to account for the interac- 
tion of computer access at home on the three home computer-use variables, and 
for the interaction of computer access at school with the two school computer- 
use variables. Finally, Fuchs and Woessman calculated the effect on math and 
reading scores with all elements added into the model: home access and use; 
school access and use; and all the control variables. 

WHAT EXACTLY DID THE STUDY FIND? 

Fuchs and Woessman estimated that students with a computer at home would 
score about 23 points higher in math and 22 points higher in reading. That is with- 
out taking into account any other characteristics. The advantage was even greater 
(29 and 26 points respectively for math and reading) for students with more than 
one computer at home. These differences were significantly greater than 0 (p>.01). 
To give an idea of the magnitude of this effect, the PISA test mean is 500 and the 
standard deviation is 100. The advantage for computer access (uncorrected for any 
other effects) is thus about .22-.29 standard deviation units. According to Fuchs 
and Woessmann, it is almost as much as the difference in mean scores between 
grades 9 and 10, the school levels that include most of the 15-year-olds in the study. 

Accounting for the effects of student characteristics (age, gender, grade) made 
only a small difference in results; however when family background variables 
were added to the model (essentially subtracting the advantages attributable to 
wealthy, stable, supportive parents), the unique advantage of home computer 
access fell dramatically. A similar pattern was found when the analyses were re- 
peated for school computer access. 

The computer-use regressions, which included all the control variables, 
showed mixed results, as detailed below. However, (with changes in magnitude 
on the order of a few points) the pattern of effects for home access and use re- 
mained stable when effects for school access and use were added into the model. 
That is, there were no apparent major interactions between these variables. 

After removing effects of family background and school characteristics, Fuchs and 
Woessmann found these relationships in the PISA data on technology and test scores: 

• Students who reported having educational software at home had an 
advantage of about two points in math (p<.05) and no significant dif- 
ference in reading. 
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• Students who reported having a link to the Internet at home had an 
advantage of about five points in math and about four points in read- 
ing (p<.01). 

• Students who reported having one computer at home had a disadvan- 
tage of about seven points in reading and about six points in math. 
Students who reported having more than one computer at home had 
a disadvantage of about 16 points in reading and about 14 points in 
math (p<.01). 

• Students whose principals felt that instruction was hindered a lot by 
lack of computers had no significant advantage in math and reading. 
Nor did students whose principals felt that instruction was not hin- 
dered at all by lack of computers. There was no significant relationship 
between test scores and computer density. 

• Students who reported never reading “e-mail and Web pages because 
(they) want to” had a disadvantage of seven points in math and about 
six points in reading. Students who reported reading email and web 
pages several times a week had an advantage of about six points in 
math and about nine points in reading (p<.01). 

• Students who reported “never or hardly ever” using computers at 
school had about a two-point disadvantage in math (p<.05) and no sig- 
nificant difference in reading. Students who reported using computers 
several times a week had a disadvantage of about six points in math and 
five points in reading (p<.01). 

• Students who reported never or hardly ever using the Internet at school 
had a disadvantage of about two points in math and reading (p<.05 in 
math and p<.10 in reading). Students who reported using the Internet 
at school several times a week reported a disadvantage of about nine 
points in math and 13 points in reading (p<.01). 

The last two findings are conditional; that is, students who reported interme- 
diate levels of use between “never” and “several times a week,” actually had an 
advantage in math and reading. 

WHAT DOES THE STUDY MEAN? 

Finally, we get to the question of how to interpret these findings such that 
they can be used for policy or practice. This is how Fuchs & Woessmann state 
their central conclusions: 

. . . once family background and school characteristics are extensively 
controlled for, the mere availability of computers at home is negatively 
related to student performance in math and reading, and the availability 
of computers at school is unrelated to student performance. By con- 
trast, student performance is positively related to the use of computers 
at home for accessing e-mails and Web pages and to the availability 
of educational software at home. Finally, student performance shows 
an inverted U-shaped relationship with the extent of computer and 
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internet use at school, rising with some use but falling again with a use 
of several times a week (Fuchs & Woessmann, 2004, p. 17). 

The key phrase in this statement is “extensively controlled for.” The impetus 
for the reanalysis of PISA data was skepticism by Fuchs and Woessman toward 
the apparent dramatic effect of having access to computers. From the stand- 
point of the educational technology research, this skepticism is well founded. 

Mixed findings for overall effects of technology use are the norm. This has 
been the case in other correlational analyses of large data sets, such as such as 
Wenglinsky’s (1998) analysis of National Assessment of Educational Progress 
scores, and Goolsbee and Guryahs (2002) study of e-Rate subsidies in Califor- 
nia. Research reviews have found the same issues in other types of studies, along 
with a deplorable lack of controls (Fours, 2000; Waxman, 2002). 

On the other hand, there is also ample evidence that the use of technology 
can have a significant positive effect on student learning. James Kulik’s (2003) 
meta-analysis of controlled studies found effect sizes of .25-. 36 standard devia- 
tion units for applications including integrated learning systems, writing, and 
tutorials. The U.S. Department of Education’s What Works Clearinghouse, in a 
review of middle school mathematics programs, found only two studies that not 
only showed positive results but also met the very rigorous WWC standards for 
experimental design. However, both of those addressed interventions with sub- 
stantial technology components (What Works Clearinghouse, 2004). 

What makes the difference in findings? Fuchs and Woessmann have part of the 
answer: control of variables. Studies with significant positive effects for technol- 
ogy tend to be those with well-defined interventions that can specify the condi- 
tions of implementation (e.g., Boster, 2002, 2004; Morgan & Ritter, 2001). 
There are numerous variables affecting successful implementation, including 
where in a building computers are used, how they are scheduled, the breadth 
versus depth of the curriculum, the computer knowledge of teachers, and teach- 
ers’ pedagogical approaches (Becker & Ravitz, 2001). In the United States, the 
National Educational Technology Standards specify a number of conditions and 
competencies that teachers in a Delphi-type consensus process have agreed are 
important for integrating technology in teaching and learning (International 
Society for Technology in Education, 2000, 2002). The instrumentation for as- 
sessment at this level of detail is well-established. Self-report surveys, observation 
protocols, and other approaches for assessing technology attitudes and use have 
been widely reported on in this and other journals (International Society for 
Technology in Education, 2003; Johnson & Barker, 2002; Knezek, 1998). 

The point here is that, from the standpoint of educational technology re- 
search, the proxies (Fuchs and Woessman’s term) for technology access and use 
are not adequate to define the constructs. Or, put another way, “school charac- 
teristics” were not “extensively controlled for,” to the extent that those charac- 
teristics include the instructional context of technology use. It is interesting to 
note that there was an optional survey on students’ computer interests admin- 
istered in about two-thirds of the countries participating in PISA 2000 (Adams 
& Wu, 2002). This survey dealt with some of the issues covered in typical 
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educational technology research (e.g., student use of particular types of applica- 
tions). However, Fuchs and Woessman did not use those data in their analysis. 

There are other critiques that might be made of the PISA data and the re- 
gression analysis of technology access and use. For instance, anyone who has 
worked with technology in schools will have encountered principals whose be- 
liefs about technology use in the building vary widely from the expressed beliefs 
and observable behaviors of the teachers. Furthermore, even a well-informed 
and instructionally astute administrator might have a difficult time giving a reli- 
able single rating for technology adequacy across all subjects and classrooms in 
a school. 

Another caveat offered by Fuchs and Woessmann themselves is that their 
work is only correlational. They can propose explanations, but they cannot test 
them without an actual controlled experiment. Others have raised concerns 
about the PISA methodology itself: the difficulty of writing “real-life” items for 
students across multiple cultures and curricula and the problem that 15-year 
olds are at different stages of schooling in different countries (Prais, 2003). 

However, the contention of this paper is that, even if the regression model 
were more complex, the sampling issues resolved, and the reliability established 
for all items, we are still left with no more than (1) the presence of technology 
is not, by itself related to student achievement and (2) the use of technology 
may help or hinder academic learning, depending on the nature of the use. The 
quality of data in a survey such as PISA 2000 simply does not permit much 
more elaboration. 

These two findings are not trivial. Similar issues have been debated among 
educators in the past, notably in the meta-analyses of James Kulik and Richard 
Clark (Clark, 2001). At this point, however, these assertions are to educational 
technology what “buy low, sell high” is to economics — fundamental principles. 
They are best expressed in the National Educational Technology Standards’ 
“essential conditions” for technology integration, which include alignment of 
curriculum, instruction, assessment, technology resources, and teacher skills 
(International Society for Technology in Education, 2000). 

Fuchs and Woessman were correct to question the simple conclusion that 
student achievement is positively related to computer access. And there will 
certainly be more opportunities to consider these relationships in databases 
coming out of other large-scale assessments. (The preliminary results from PISA 
2003 were released at about the same time as Fuchs and Woessman published 
their re-analysis of the 2000 data.) The challenge for the technology-using edu- 
cators is to make clear to the headline writers — and those who consume the 
headlines — that the predictable rediscovery that technology’s effect depends on 
curriculum and instruction is not the revelation of a secret, but the affirmation 
of well-known principles of teaching and learning. 
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